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Dynamic  fTogramming  unc  a  New  Formalism  in  the  Calculus  of  Variations 

By 

Richard  Sal  Lman 

1.  Introduction.  In  a  series  of  papers,  ,  wa  have  treated  a 

number  of  mathematical  problems  arising  from  multi-stage  decision  pro¬ 
cesses.  Problems  of  this  type  occur  in  tne  theory  of  probability,  cf. 

Ll,  [.♦],  L?7»  ^3  ;  in  mathematical  economics,  DJ,  £>3,  G3  ,  CtJ5  ["] i 

in  control  processes,  Wj  [&J  p  in  learning  processes,  [/],  ;  and 

in  many  other  fields  as  well. 

ln'4hle  pa  per ,  we  »i«h  to  show,  that  the  functional  equation  tech¬ 
nique  introduced  in  Vha  worr.s  may  be  used  to  provide  a  n6w  ap¬ 

proach  to  some  classical  problems  in  the  calculus  of  variations,  in 
addition  to  furnishing  a  new  analytic  weapon,  feel  that*  the  method 

hbs  greht  potentialities  as  a  computational  tool.  — ' 

»  ■ — — 

As  we  have  pointed  out  previously,  L*}  •  this  a  j  p  r  ou ch 

seems  ideally  suited  to  the  handling  of  variational  problems  involving 
stochastic  processes.  This  point  will  be  further  enlarged  upon  in  some 
forthcoming  publications. 

To  illustrate  the  approach  in  its  simplest  setting,  we  shall  consicer 

first  the  problem  of  maximizing  $  F(x,z<du,  suLject  to  the  constraint 

0 

dx/du»  U(x,zj,  x(0)  s  c,  where  the  analytic  cetall  is  at  a  minimum,  ^nd 
tnen  turn  to  the  eigenvalue  problem  derived  from 

—  *  ♦(.*)  Iso.  *  *(•)  =  O  .  ,  , 

JUtv  *  *  [l  A  i 

which  is  rather  less  straightforward  . 


1 


In  both  cases,  we  shall  derive  a  partial  differential  equation  for 
the  desired  quantity.  The  method  W6  sketch  below  is  also  applicable  to 
multi-dimensional  eigenvalue  problems  whenever  the  underlying  apace  has 
certain  symmetry  properties,  further  results,  together  with  a  detailed 
exposition  of  the  results  contained  herein  will  appear  in  another  pub¬ 
licist!  on. 

Z.  lhe  L'.ax  1  xi/ation  rr  ob ,  ea; 

net  us  now  consider  the  maximizetior  problem  described  in  the  first 
section.  We  3hall  consistently  assume  that  all  iunctions  that  up;  ear  pos¬ 
sess  ail  tae  differentiability  properties  required  to  matte  our  operations 
legitimate,  since  v«e  :or  e  primarily  interested  here  in  presenting  the  basic 
formalism. 

The  essence  of  our  method  xies  in  considering  the  maximum  value  as 
a  function  of  the  parameters  whica  describe  the  state  of  the  process, 
the  "state  variables",  in  tnis  cas6,  these  are  c  and  t.  We  write 

Max  \  r(x,zjdu  *  fic.tj  12.1; 

x  c  t 

The  classical  approach  regards  c  and  t  as  fixed  and  considers  l  r(x,zjdu 

O 

as  a  functional  of  z.  in  contrast,  .ve  regurc  c  und  t  as  parameters  and 
seer:  to  determine  zioj  as  a  function  of  c  and  t. 

opeaKir.g  in  ttrms  of  u  multi-stage  process,  in  place  of  determining 
the  optimal  continuation  from  on6  fixed  j  os  it  ion,  v/e  try  to  find  the  opti¬ 
mal  first  step  from  any  position. 

net  us  proceed  to  the  analysis  which  will  clarify  the  above  general 
remarks.  We  ve  ^  t 

4(c,wr)=  ^  *  M**  [  S  FK»)Ut  ^  PbjDlu. 

o  1  ' 

12.2/ 


it  i 3  clear  that  whatever  the  choice  of  ziu)  in  U  u  i  s,  if  we 

wish  to  maximize  the  right  sice  of  i£.2»  we  must  choose  ziuj  in  a$  u  « 

x*  v 

s*  t  so  as  to  maximize  ^F(x,z)du  subject  to  the  constraints  dx/dt* 
l>|x,zi  ,  xis)c  cis).  here  ci  3)  is  tne  new  initial  value  obtained  from  the 
differential  equation  dx/dt  *  G( x,zj  ,  xiO)  *  c. 


In  view  of  the  invariant  nature  of  the  problem  with  respect  to 


translations,  we  see  tnat  we  m-st  have 
5  F( x , z;du  *  ficlsj ,t; 


(2.3 ) 


Hence  (2.2/  yields  the  fund. mental  functional  equation 

f  ( c ,  st  t  j  x  Max  ^r(x,z)dut  fl c l s  j ,  t  j  1  .  I  2 .  i*  j 

The  notation  zto.sj  indicates  tnat  th*  maximization  is  over  all  z(u) 
taxer,  over  G4u*s.  as  s  shrinks  to  C,  the  choice  of  a  function  reduces 
to  a  cnoice  of  zlG),  a  quantity  whicn  we  snail  call  w  for  typographical 
convenience,  n  straightforward  calculation  yields 


K.ax  [  i*  ( c  ,  w  4  t  u(c#w,fe] 


(2.5 


ns  the  limiting  f_rm  of  1 2.Uj  js  s  -*  u.  The  maximizing  w  is  obtained 
by  e  ,uatlng  the  partial  derivative  with  respect  to  w  to  zero, 

0  *  r\v  t  uw  Joabinln  ;  this  with  t2.5 we  obtain  the  system  of  equations 

3  G-Pw>/6-w  s  K(<.w) 

"  ""  j  G*w  «■  ^  (c->*0  1 2 . 6 ; 


To  derive  a  pertial  differential  equation  for  w,  v;e  employ  the  identity 


/f  \  s  fi  \  .  The  rosult  is 

'  T  /l  *  /  X 


wt  =•  Lw 


1 2 .7) 
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As  is  well -luiown,  cl'  Qx)  ,  the  general  solution  of  tnis  equation 

may  te  obtained  from  the  general  solution  of  a  system  of  first  order 

ordinary  differential  equations,  aince  c  is  known  for  t*G,  to  complete 

tns  solution  we  require  only  the  value  of  w  at  t*0.  ror  small  t  we  have 
■c 

^F(x,z;du  »  Flc,w)t  t  12.8; 

ilence  w  at  t  *  U  is  determined  by  F  *  0  . 

4  w 


Taking  o(x,zj*z,  we  are  reduced  to  the  familiar  problem  of  maxi¬ 
mizing  ^  rlx,x')du.  it  is  readily  verified  thr-t  the  characteristic  equa¬ 
tions  obtained  from  (1.7 1  are  equivalent  to  the  usual  rsuler  equation. 


Th6  general  problem  of  maximizing  5 
subject  to  tne  constraints  dx ;  /dt -  U ; v x  , z ; , 
in  tne  some  fashion. 


■>  idu 
x„iC;  *  c^  ,  may  be  attacked 


3.  ihe  a if  e rival  u6  irsLea. 

The  eigenvalue  problem  posed  in  i  1  is  equivalent  to  that  of  da- 

f  *  |  ^ 

t.rmir.ing  the  successive  minima  of  }  *  JUv  subject  to  the  constraints 

0 

,*  V 

j  Sx  x.(t)  *  x.0)  =  O  .  TO  treat  the  problem  using  the  functional 
equation  approach,  w6  imbed  it  within  th6  more  general  problem  of  deter- 

r-V 

a.Lning  the  minima  of  «j(u;  •  }  n>  JUc  subject  to  the  constraints 


(*) 

*  »  Cat  t)  & 

o 

v 

o  $U>x  i*  -*  Jift 

Okr 

\  (.Stt-*)  ZAm.  *  /% 

a. 

1  3. 

1) 

:ir. 

Jiu;  *  fia,k,t;. 

Then,  if,  as  above,  we  write 

its 

r  fttstr 

* 

^*>*<•*0  *  \ 

X*  s 

o. 

we  cannot  derive  a  junctional  equation  immediately,  since  uia*s;  la  not 
necessarily  zero,  io  simplify  the  analysis,  let  us  pass  directly  to  the 


* 


derivation  of  the  partibl  differential  e  ,uation,  assuming  that  a  is 
an  infini t«a)Mi  o  It  is  sufficient  to  consider  only  terras  which  are  of 


zeroeth  or  first  order  in  s.  since  r  sy'(a)  ,  we  set 

“  i(u)  X-<0/t  •  Then  (o •*<,■>  t)  =  c  »  to  the 

,  a 

order  of  our  aj^roximation,  and  x‘(v)-  V(n)*-  su.t*)/t  .  rnus ,  to  term3  in  s 


c1*4'r  v  C 

*  (•k*  ***  3  (atit  r-u^  ^ 

'  »♦* 


.  The  constraint  of  13.1b/  becomes 

a*ytr 


-  s  x(»\ t)  O  (V) 


13.3) 


w  .ie  T6 


a*  t 


4(<0 


l  3.4/ 


UaKlng  t  change  of  variable  ^  ,  to  reduce  the  constant 

term  in  13*3 1  to  1,  the  constraint  twites  the  form 
o  «■«  *t  «•»  $•»  r 


o*i 


L  ■***  S  \1T  *  (*'>'* -*)U  *  I+Ctil),  13.3/ 


s 


4  W  w  >V  r 


The  e  nation  in  i3.</  becomes 

a*  a 


Tw  [  ^  *  0(s>)  o.fci 

*'[■-, *■ 


netting  s o,  tnis  yields  the  partial  a  ifl  er6  nt  ia  1  6  nation 

\  =  [_  •*•  ** f  (.  Att 

Since  \>  r  i  where  is  inc-penuent  of  i'ia|,  the  uni.,u6  minimum 


occurs  wnen 


1»'U)  +  [  r  ~-)^M  -  i*  *  o  '3.8; 

eliminating  x'va ;  by  means  of  this  last  relation  we  ootain  a  nonlinear 
partial  differential  equation  for  f. 

The  C6navior  of  f  for  small  t  a«y  be  determined  by  solving  the 
problem  of  minimizing  ^  x1'’!*.  subject  to  the  constraints 

xia)r  xia*t)  s  0  V  3  -  9 ) 

a*  t  9*t 

$  la)  \  I’**,  t  -*  \  *>lu)  C6*t-u)xlu-  I 

This  problem,  in  turn,  muy  be  treuted  by  the  partial  differential  equa¬ 
tion  approach,  using  the  fact  that  th6  solution  for  k«G  is  immediate. 
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f.  A  similar  formalism  has  b-en  applied  by  P.  Isaacs  in  his  theory 

of  differential  games,  i.e.  systems  of  differential  equations 

of  the  form  dx*/dt  *  Min  Max  F.(x,z,w). 
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